l METHOD OF TRANSMITTING LAYERED VIDEO-CODED INFORMATION 

2 
3 
4 

5 Technical Field 

6 Embodiments of the present invention relate to transmitting video 

7 information over a network. In particular, embodiments of the present 

8 invention relate to methods for setting a transmission priority for layered 

9 video-coded information that is to be transmitted over a network. 

10 

n Background of the Invention 

12 One of the most promising applications in the field of communications 

13 is the transmission of encoded or compressed video information across 

14 mediums such as telephone networks, cable television networks, and the 

15 Internet. The video information is typically encoded at one location, 

16 transferred over a network to a destination, and decoded at the destination, 
n A video encoder may be used to, among other things, receive video signals 

18 and other data, convert these to digitized form (if the data is not already in 

19 digitized form), and compress the data using an encoding algorithm. 

20 Examples of video encoding algorithms are the MPEG-1, MPEG-2, and 

21 MPEG-4 digital video-coding standards of the Motion Picture Expert Group 

22 and the H.261 and H.264 standards from the ITU-T (International 
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1 Telecommunications Union-Telecommunications sector). The video 

2 information may be first obtained from a video camera that generates video 

3 signals which contain a series of video frames that each depict images. In 

4 addition, the video-coded information may also include other components 

5 such as, for example, animations, text, and/or audio information. A video 

6 decoder may be used at the destination to decode the received video-coded 

7 frames into a format that may be displayed by a video monitor or other 

8 device. 

9 The networks over which video-coded information is transmitted 

10 generally have limited bandwidth. The amount of bandwidth available at any 
l i given time depends upon factors such as the amount of traffic and the 

12 operability of network components. The amount of available bandwidth is a 

13 major parameter for the quality of the video presented by the video decoder, 

14 with an increase in bandwidth increasing the quality of the video. The 

15 unavailability of bandwidth may cause video-coded frames to be lost or 

16 delayed while being sent across the network due to network congestion or 
n buffer overflow. Such losses and delays may cause severe degradation in 

18 perceived quality of the video displayed by the video decoder and may result 

19 in error propagation, especially where the video is sent in real-time or 

20 streamed or where the video contains objects in motion. 

21 One way to combat the impact of limited bandwidth is to use a 

22 "layered" video-coding scheme. Layered video-coding schemes divide a 

23 stream of video-coded frames into layers. In the standard layered video- 

24 coding scheme, the video-coded information signal is divided into a base 

25 layer and an enhancement layer. The enhancement layer typically provides 

26 refinement information that is added to the base layer to provide higher 

27 quality. For example, the base layer may typically be decoded and 

28 displayed without any data from the enhancement layer, although this would 

29 probably result in reduced picture quality. When network congestion occurs, 

30 packets may be dropped from the enhancement layer, but valid video at a 

31 slightly lower quality may still be produced by decoding packets from the 

32 base layer. 
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1 Traffic that is sent over a network is often classified by a priority level. 

2 In many networks, the priority level will be used if the network needs to drop 

3 some of the packets being transported. For example, if network congestion 

4 occurs and the network has to drop packets, the network will typically drop 

5 the lower level packets. A video encoder may take advantage of these 

6 classifications by designating some video-coded frames as high priority and 

7 the rest as low priority. For example, every fifth frame may be sent as high 

8 priority, with the rest being sent as low priority. A typical layered video- 

9 coding scheme is to transmit enhancement layer traffic over the network with 

10 a lower priority than the base layer. Another scheme, which is referred to as 
n using multiple threads, codes the base layer frames independently, using 

12 only previous base layer frames for motion compensation, and codes the 

13 enhancement layer frames normally, using both base and enhancement 

14 layer frames for motion compensation. 

15 A problem may occur when using a layered video-coding scheme if 

16 there are simultaneous attempts to transmit high priority base layer frames, 
n If this occurs, the network may become overloaded with high priority data. In 

18 this case, either base layer frames may be lost, causing drastic quality 

19 reduction, or the network may be forced to buffer some of the base layer 

20 frames, causing excessive delays. 

21 
22 

23 Summary of the Invention 

24 Embodiments of the present invention alleviate degradation in a 

25 layered video-coded system by using feedback from the network indicating 

26 whether or not capacity is available for sending the high priority frames. The 

27 feedback may include a response to a request for information on whether the 

28 network currently has the available capacity to transmit additional high 

29 priority traffic. In an embodiment, a candidate base layer frame is 

30 transmitted over the network as a base layer frame if permission to send 

31 high priority data was granted and is transmitted over the network as an 

32 enhancement layer frame if permission to send high priority data was 
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1 denied. In a further embodiment, a candidate base layer frame is deleted if 

2 permission to send high priority data was denied. 

3 
4 

5 Brief Description of the Drawings 

6 FIG. 1 is a block diagram of a system including a video encoder and 

7 decoder according to an embodiment of the present invention. 

8 FIG. 2 is a block diagram of a video encoder according to an 

9 embodiment of the present invention. 

10 FIGS. 3-7 are flow diagrams of methods of transmitting video 
n information according to embodiments of the present invention. 

12 

13 Detailed Description of the Invention 

14 The present invention provides an improved method of transmitting 



15 layered video-coded data by using network feedback which indicates 

16 whether bandwidth is available to send high priority frames (e.g., base layer 

17 frames). In an embodiment, the encoder does not decide a priori which 

18 frames are to be transmitted as base layer frames, but rather requests 

19 permission from the network to transmit high priority data before encoding a 

20 candidate base layer frame. In this embodiment, the candidate base layer 

21 frame may either be encoded and transmitted as a base layer frame if 

22 permission to send high priority data is granted or may be transmitted as a 

23 low priority frame, or deleted from transmission, if permission to send high 

24 priority data was denied. This and other embodiments are described in more 

25 detail below. By using an embodiment of the present invention, the 

26 likelihood of network overload is minimized because the arrival of high 

27 priority data into the network is randomized. In addition, embodiments of the 

28 invention avoid buffering large amounts of data and thus avoid the delay that 

29 such buffering may cause. Use of the present invention may therefore 

30 alleviate degradation of perceived picture quality due to loss of frames or 

31 buffering delays. 
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1 FIG. 1 is a block diagram of a system including a video encoder and 

2 decoder according to an embodiment of the present invention. FIG. 1 shows 

3 a video encoder 1 1 0 and a video decoder 1 20 both of which are coupled to a 

4 network 130. The term "coupled" is intended to encompass and be broader 

5 than the term "directly connected." As discussed below, encoder 110 or 

6 another device may transmit video-coded data (such as video-coded frames 

7 1 13) to decoder 120 across network 130 at a priority level that is determined 

8 using information about the network's available capacity (such as network 

9 capacity information 116). This information may be provided in response to 

10 a request for such information (such as network capacity request 115). Of 
n course, the present invention is not limited to the system shown in FIG. 1. 

12 For example, in other embodiments, video-coded data may be sent by an 

13 encoder such as encoder 110 to devices other than a decoder. For 

14 example, encoder 110 may send video-coded data to a data storage facility 

15 for later use by a decoder. As another example, video-coded data may be 

16 sent from encoder 111 to a transmission device which transmits the data 
n across the network. Similarly, in other embodiments, stored video-coded 

18 data that had been previously encoded may be transmitted to a decoder 

19 such as decoder 1 20 by a device other than an encoder. 

20 Network 130 may be any information systems network across which 

21 information can be sent. Examples of network 130 include the Internet, an 

22 intranet, a wireline telephone network, a wireless telephone network, a 

23 virtual private network, or any combination of these networks. Encoder 110 

24 may be used to encode video data, such as video signals 112, for 

25 transmission across a network as video-coded frames. Encoder 110 may 

26 output video-coded data that complies with a video-coding standard or 

27 algorithm such as MPEG-1, MPEG-2, MPEG-4, H.261, and H.264. In FIG. 

28 1 , encoder 1 1 0 is shown coupled to a video camera 111. In an embodiment, 

29 video camera 111 outputs video signals 112 to encoder 110. In other 

30 embodiments, encoder 110 may encode prerecorded video signals or may 

31 receive other types of input such as animation, text, or audio data. Decoder 

32 120 may be used to receive and decode video-coded frames for display as 
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1 video images. Decoder 120 is shown coupled to monitor 121 and may 

2 output decoded video data, such as video signals 122, for display as video 

3 images on monitor 121 . 

4 As shown in FIG. 1, network 130 includes a network access server 

5 131 that connects encoder 1 10 to network 130 and a network access server 

6 132 that connects decoder 120 to network 130. Network access server 131 

7 and network access server 132 may permit connections to network 130 from 

8 outside the network and may be operated by a third-party such as AT&T 

9 Corp.® or America Online, Inc.® Network 130 includes a network 

10 administrator 133 which administers operation of network 130. In an 
n embodiment, network 130 may be decentralized and may contain multiple 

12 network administrators. Fig. 1 shows a high priority channel 134 and a low 

13 priority channel 136 coupling network access server 131 to network access 

14 server 132. In an embodiment, high priority channel 134 and a low priority 

15 channel 136 are logical paths across network 130, and data may be 

16 transmitted across each logical path using various physical paths. In an 

17 embodiment, network 130 contains a plurality of virtual circuits that may be 

18 used to transmit high priority traffic and a plurality of virtual circuits that may 

19 be used to transmit low priority traffic. In an embodiment, network 130 

20 contains a plurality of nodes, such as router or switches, and many of these 

21 nodes may be used to transmit data for both high priority channel 134 and 

22 low priority channel 136, 

23 A person of skill in the art would appreciate that the amount of traffic 

24 across network 130 may be dynamic, and thus that the amount of available 

25 bandwidth may be constantly changing over time. At any given time, 

26 network 130 may experience congestion and may be unable to provide 

27 bandwidth for transmitting additional data. When network congestion 

28 occurs, network 130 may take appropriate actions such as dropping data or 

29 buffering data. For example, if a data packet XYZ is sent from network 

30 access server 131 to be forwarded across network 130 to network access 

31 server 132, but network 130 does not have the bandwidth to transmit data 

32 packet XYZ, then data packet XYZ may be dropped and a message 



370493 v1 



-6- 



2685/5737 KRC 



1 indicating this dropping may be transmitted back to encoder 110. In an 

2 embodiment, network 130 will drop low priority packets before dropping a 

3 high priority packet. Network 130 may impose an additional charge for the 

4 transmission of data on a high priority channel. 

5 In an embodiment, network administrator 133 maintains information 

6 about the amount of currently available network bandwidth and may provide 

7 information about current network capacity and expected future network 

8 capacity as feedback to encoder 110. For example, encoder 110 may send 

9 a network capacity request 115 through network access server 131 to 

10 network administrator 133, and in response network administrator 133 may 
n send network capacity information 1 1 6 through network access server 131 to 

12 encoder 110. Network capacity request 115 may be a request for 

13 information about the availability of capacity to send a type of data (e.g., a 

14 high priority frame or base layer frame), and network capacity information 

15 116 may be a response to such a request. In an embodiment, a request for 

16 information sent from encoder 110 does not specify the type of data to be 
n transmitted (e.g., that the data is video-coded frames) and may be, for 

18 example, a request for a specified amount of bandwidth. In this example, 

19 the priority of the.bandwidth, the size of the data, and/or the duration may be 

20 specified. In an embodiment, a request is not required for the sending of 

21 network capacity information 116, and network capacity information may be 

22 sent, for example, on a periodic basis or during conditions determined by 

23 network administrator 133. Of course, a person of ordinary skill in the art 

24 would appreciate that network capacity request 115 and network capacity 

25 information 116 may be transmitted between encoder 110 and network 

26 access server 131 using the same connection as the video-coded frames 

27 113. Embodiments of the present invention that make use of this network 

28 capacity information when transmitting video-coded information are further 

29 described below. 

30 FIG. 2 is a block diagram of a video encoder according to an 

31 embodiment of the present invention. FIG. 2 shows an encoder 110 such as 

32 in FIG. 1 above. In addition, FIG. 2 shows a stream of video signals 1 12 that 
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1 is coded by encoder 1 1 0 and transmitted as a stream of video-coded frames 

2 113 across network 130 over two logical channels (high priority channel 134 

3 and low priority channel 136). FIG. 2 shows six video frames that may be 

4 part of the same stream of video data. The first four frames in the stream 

5 (i.e., 1 st video frame 221 , 2 nd video frame 222, 3rd video frame 223, and 4th 

6 video frame 224) have already been encoded and are being transmitted over 

7 network 130, while the other two frames (5 th video frame 225 and 6 th video 

8 frame 226) consist of uncoded data that is being transmitted for coding to 

9 encoder 110 by, for example, video camera 1 1 1 of FIG. 1 . Of course, each 

10 of the video-coded frames may comprise multiple frames of uncoded video 
n data. As discussed above, the video-coded frames may also include non- 
12 video information such as animations, text, control information, and/or audio 

13 information. In the embodiment shown, each of the video-coded frames 113 

14 transmitted across network 130 contain a priority level field. A frame that 
is has a low priority level such as 2 nd video frame 222 is transmitted on low 
16 priority channel 136. In an embodiment, the priority level is built into the 
n video-coded frame when it is encoded. In other embodiments, the priority 

18 level may be added to a video-coded frame after the frame is encoded or 

19 may be transmitted to network 130 as a data element separate from the 

20 video-coded frame. 

21 As shown in FIG. 2, encoder 110 contains encoding logic 117 and 

22 transmission priority setting logic 118. Encoding logic 117 may be used to 

23 encode the video data. Transmission priority setting logic 118 may send 

24 network capacity requests 115 and may receive network capacity 

25 information 116 in response to these requests. Transmission priority setting 

26 logic 118 may use the network capacity information to select a transmission 

27 priority for each of the video-coded frames 1 1 3 according to an embodiment 

28 of the present invention such as the method discussed below. Of course, 

29 encoder 110 may contain additional components such as logic to control the 

30 transmission of data to network 130. As would be appreciated by a person 

31 of ordinary skill in the art, encoder 1 10 may also contain an analog-to-digital 

32 converter, transform logic, quantizer logic, and prediction logic. The 
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1 components in encoder 110 may be embodied in software, hardware, 

2 firmware, or some combination. Encoder 110 may include one or more 

3 central processing units (CPUs), random access memories (RAMs), and 

4 Read Only Memories (ROMs), and memories in encoder 130 may store 

5 instructions to be executed by a processor to carry out embodiments of the 

6 invention as described herein. The logic in encoder 110 may be embodied 

7 in a special purpose processing unit such an application specific integrated 

8 circuit (ASIC). In an embodiment, encoder 110 is a personal computer (PC) 

9 that contains software to carry out an embodiment of the present invention. 

10 FIG. 3 is a flow diagram of a method of transmitting video information 
n according to an embodiment of the present invention. The description below 

12 is explained with reference to the systems shown in FIGS. 1 and 2, but of 

13 course these methods may be used with other systems. As shown in FIG. 3, 

14 a frame of video data to be coded as video-coded data is received (301). 

15 For example, a frame of video data may be received at encoder 110 from 

16 video camera 111 or may be received from a buffer in another part of 

17 encoder 110. In embodiments, the frames of video data may already be 

18 stored at the encoder (e.g., buffered at the encoder) prior to coding. The 

19 encoder may send a request for information relating to the network's 

20 available capacity to transmit high priority data (302), and capacity 

21 information may be received in response to the request (303). Examples of 

22 the request and response are shown in FIG. 1 as network capacity requests 

23 115 and network capacity information 116. In an embodiment, network 

24 administrator 133 will determine if the capacity to send high priority data is 

25 available and will send a response to encoder 110. If the response indicates 

26 that the network has the capacity to transmit the frame as high priority data 

27 (304), then the frame is encoded as a high priority frame (e.g., a base layer 

28 frame) (305) and is transmitted as high priority data (306). If the response 

29 indicates that the network does not have the capacity to transmit high priority 

30 data (304), then the frame is encoded as a low priority frame (e.g., an 

31 enhancement layer frame) (307) and is transmitted as low priority data (308). 

32 Frames that are transmitted as high priority data may use a high priority 
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1 channel across network 130 and frames that are transmitted as low priority 

2 data may use a low priority channel across network 1 30. 

3 FIG. 4 is a flow diagram of a method of transmitting video information 

4 according to another embodiment of the present invention. As in the 

5 previous embodiment, the encoder may receive a frame of video data to be 

6 coded as video-coded data (401), may send a request for information 

7 relating to the network's available capacity to transmit high priority data 

8 (402), and may receive capacity information in response to the request 

9 (403). In this embodiment, the data is video-coded (404) and the video- 

10 encoded frame is buffered (405) at substantially the same time as the 
n request is sent and/or the response to the request is received. If the network 

12 has the capacity to transmit the frame as high priority data (406), then the 

13 request is transmitted as high priority data (407). If the network does not 

14 have the capacity to transmit high priority data (406), then the video frame is 

15 transmitted as low priority data (408). In some embodiments, such as very 

16 high compression codecs, the frame has to be receded as a high priority 

17 frame. In other embodiments, the frame may have been coded at both 

18 priorities (high and low) in parallel, and the frame with the desired priority 

19 may be chosen at a later time. 

20 FIG. 5 is a flow diagram of a method of transmitting video information 

21 according to another embodiment of the present invention. FIG. 5 contains 

22 steps 401 to 407 as in FIG. 4. In this embodiment, however, the encoded 

23 frame is deleted from transmission (500) if the network does not have the 

24 capacity to transmit high priority data. When an encoded frame is deleted 

25 from transmission, the video data may be lost, but such a loss may not 

26 greatly impact perceived picture quality as long as the encoder 1 10 is aware 

27 of the loss. 

28 FIG. 6 is a flow diagram of a method of transmitting video information 

29 according to another embodiment of the present invention. As discussed 

30 above, a frame of video data to be coded as video-coded data may be 

31 received (601). In this embodiment, the frame is then encoded as a high 

32 priority frame (602). In this embodiment, all frames may be encoded as high 
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1 priority frames prior to requesting and/or receiving any information about 

2 available network capacity. As shown in FIG. 5, after encoding the high 

3 priority frame, a request for information relating to the network's available 

4 capacity to transmitting high priority data is sent (603), and capacity 

5 information is received in response to the request (604). If the network has 

6 the capacity to transmit the frame as high priority data (605), then the 

7 request is transmitted as high priority data (606). If the network does not 

8 have the capacity to transmit high priority data (605), then the video frame is 

9 transmitted as low priority data (607). 

10 FIG. 7 is a flow diagram of a method of transmitting video information 
n according to another embodiment of the present invention. A frame of video 

12 data to be coded as video-coded data may be received (701). In this 

13 embodiment, frames are encoded as either high priority frames or low 

14 priority frames based on a priority selection algorithm (702). The algorithm 
is may use information about how much bandwidth is allocated for the 

16 transmission of high priority frames. The encoder then obtains information 

17 relating to the network's available capacity to transmit high priority data 

18 (703). This information may be obtained by network feedback or other 

19 means such as the Real Time Control Protocol (RTCP) from the Internet 

20 Engineering Task Force (IETF). If more than a threshold number of low 

21 priority frames have been lost (704), then an additional number of the frames 

22 may be encoded as high priority frames than is dictated by the transmission 

23 priority algorithm discussed above. These additional high priority frames 

24 may be encoded at a lower quality than is generally used for high priority 

25 frames. 

26 The present invention improves the transmission of video information 

27 by using network feedback on available bandwidth. Through the use of 

28 feedback the present invention allows for the randomization of the arrival of 

29 high priority data into the network and, therefore, minimizes the likelihood of 

30 network overload. The present invention may also avoid the buffering of 

31 large amounts of data and the ensuing delay that such buffering may cause. 

32 Several embodiments of the present invention are specifically illustrated 
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1 and/or described herein. However, it will be appreciated that modifications 

2 and variations of the present invention are covered by the above teachings 

3 and within the purview of the appended claims without departing from the 

4 spirit and intended scope of the invention. For example, unless otherwise 

5 specified, the order of the steps performed may be varied. 
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